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ABSTRACT 


In  this  paper,  we  investigate  sensor  fusion  along  three  avenues:  statistical,  biological  and 
categorical.  The  first  two  approaches  are  analyzed  simultaneously  to  provide  a  precise 
and  rigorous  sensor  fusion  methodology.  The  statistical  model  currently  enhances 
Bayesian  methods  for  tracking,  and  suggests  further  application  to  target  identification 
and  fusion — involving  both  low  level  feature  extraction  and  higher  level  sensor  output 
combination.  The  biological  model  is  also  applied  to  multiple  levels  of  the  fusion 
problem.  On  the  lowest  level,  it  utilizes  biologically-inspired  results  for  improved  feature 
extraction.  On  the  higher  levels,  it  develops  biologically-inspired  agency  algorithms  for 
sensor  output  combination  and  sensor  network  analysis.  Ultimately,  we  model  the  entire 
fusion  process  with  category  theory.  Category  theory  allows  for  the  application  of 
advanced  mathematical  theory  to  fusion  analysis.  In  addition  to  using  category  theory  as 
a  modeling  tool,  in  this  paper  we  adapt  categorical  logic  via  topos  theory  to  provide  an 
advanced  framework  for  decision  fusion — initially  using  the  topos  of  graphs.  Graphs  are 
a  simpler  representation.  We  suggest  formulations  which  will  be  much  richer — toward 
the  goal  of  a  truly  robust,  reliable  and  computationally  practical  sensor  fusion  system  for 
assisted/automatic  target  recognition. 


Combined  Statistical,  Biological  and  Categorical  Models  for  Sensor  Fusion 


1.0  Introduction 


In  this  paper  we  attempt  to  develop  a  general  model  for  sensor  fusion — toward  the  goal  of 
assisted/automatic  target  recognition  (ATR).  To  accomplish  this,  we  incorporate  three  strands  into  our 
plan.  The  first  strand  consists  of  methods  based  on  biology.  On  the  most  primitive  level,  biological 
influence  accounts  for  the  basic  choice  of  elementary  feature  extractors.  On  a  higher  level,  biology 
inspires  our  study  of  agency  for  sensor  networks.  The  second  strand  is  statistical.  Throughout  our  system, 
we  plan  to  apply  standard  Bayesian  methodology,  particularly  particle  filters.  Bayesian  methods  present 
an  ideal  way  to  approximate  decisions  in  all  levels  of  the  fusion  process  “from  the  outside.” 

But  we  hope  to  push  our  model  further.  It  is  no  secret  that  higher-level  sensor  fusion,  ATR,  and  artificial 
intelligence  in  general  have  encountered  significant  difficulties  over  the  last  40  years  in  truly  and 
effectively  modeling  cognitive  activity,  as  it  occurs  in  individuals  or  in  groups  and  networks.  These 
difficulties  can  often  be  compensated  for  through  vaster  computational  power — in  effect,  performing 
exhaustive  searches  over  a  representation  space.  But  this  solution  can  only  be  a  temporary  one. 
Increasingly,  it  has  been  accepted  that  there  is  a  fundamental  flaw  in  automated  intelligent  methods.  We 
believe  that  that  flaw  stems  from  a  lack  of  true  subjectivity  in  machine  learning.  The  question  of  whether 
that  flaw  can  ever  be  overcome  is  unanswered,  and  an  attempt  to  answer  it  is  beyond  the  scope  of  this 
paper.  However,  our  research  aim  is  a  methodology  for  optimally  placing  subjective  elements  in  the 
system  to  address  the  flaw  in  artificial  intelligence  by  “humans-in-the-loop.”  This  concept  is  not  novel, 
but  we  advance  it  here  by  attempting  to  develop  a  theory  of  human/computer  and  human/network 
interaction  and  a  modeling  technique  that  maximizes  the  effectiveness  of  the  system  by  minimizing  the 
human  element  to  where  it  is  essential,  to  develop  a  theory  of  assisted  target  recognition  to  supplant  failed 
automatic  target  recognition.  As  will  be  shown  in  due  course,  we  try  to  achieve  these  aims  through 
category  theory.  In  the  following  sections,  we  will  first  describe  the  individual  elements  that  will  make  up 
our  system,  then  describe  the  system  and  relate  the  broader  implications  of  our  approach  to  sensor  fusion. 

2.0  Biological  and  Statistical  Methods 

We  adapt  biological  methods  at  both  lower  and  higher  levels.  At  the  lower  level,  biologically  based  filters 
are  used  as  feature  extractors.  For  one-dimensional  signals,  wavelet  analysis  is  appropriate.  However,  in 
two  dimensions,  wavelets  may  not  be  the  best  feature  extractor.  We  utilize  filters  based  on  the  human 
visual  cortex.  Our  first  approach  utilized  Gabor  analysis.  But  for  computational  reasons,  we  abandoned 
this  method  in  favor  of  oriented  ridgelets.  Ridgelets  are,  in  effect,  two-dimensional  filters  which  are 
wavelets  in  one  dimension.  Figure  1  shows  representations  of  Gabor  and  ridgelet  filters. 


Figure  1.  Sample  Gabor  filter  (left)  and  Ridgelet  filter  (right). 
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At  a  higher  level,  we  adapt  biology  to  sensor  fusion  through  the  use  of  agents.  Agents  have  been  a  very 
active  topic  in  computer  science  for  several  years.  They  have  also  regularly  been  used  to  model  sensor 
networks.  “An  agent  is  anything  that  can  be  viewed  as  perceiving  its  environment  through  sensors  and 
acting  upon  that  environment  through  effectors The  preceding  is  a  satisfactory  definition,  but  in  reality 
there  is  no  official  definition  of  agent  in  computer  science.  Additionally,  an  agent  is  not  required  to  be  a 
physical  entity  (for  example,  a  robot).  A  section  of  software  can  be  an  agent.  An  agent’s  rationality  is 
judged  by  how  well  it  scores  on  some  numerical  performance  measure.  This  is  of  course  an  external 
metric.  An  agent’s  perceptual  history,  the  record  of  every  datum  it  has  perceived,  is  its  percept  sequence, 
and  the  agent’s  behavior  is  determined  by  a  mapping  or  a  function  from  the  set  of  percept  sequences  to 
actions. 

Computer  agents  lack  an  essential  ingredient  for  real  agency — subjectivity.  Still,  computer  models  can 
impersonate  the  inferential  process  of  a  real  agent.  This  impersonation  is  usually  referred  to  as  machine 
learning  or  artificial  or  computational  intelligence.  The  inference  step  is  often  handled  with  Bayesian  or 
other  statistical  methods.  Unfortunately,  humans  don’t  normally  use  Bayesian  reasoning,  although  there  is 
some  evidence  that  if  the  problem  is  phrased  in  a  form  that  brings  out  the  Bayesian  context,  humans  are 
more  likely  to  reason  in  a  Bayesian  form.  Very  complicated  group  activity  can  be  modeled  by  computer 
agents  through  achieving  the  outward  appearance  of  inference  and  inductive  reasoning  in  individual 
agents  by  careful  mathematical  processes.  However,  the  learning  and  reasoning  techniques  for  group 
agency  are  similar  to  the  techniques  used  for  individual  agents.  Figure  2  shows  a  rough  diagram  of  our 
agency  model,  in  3  levels. 


We  have  chosen  particle  filters  as  our  prime  statistical  method,  used  in  tracking.  More  generally,  we  will 
use  Monte  Carlo  sampling  generally  for  our  statistical  analysis  as  later  we  incoiporate  Bayesian  classifiers 
into  agents’  classifying  schemes.  First  in  the  tracking  case,  state  x  at  time  t  is  updated  according  to  a 
function  of  x  and  a  weight  function  of  x.  Observed  state  y  is  a  function  of  the  true  state  x  and  a  noise 
function: 


xt+i  =ft(xt)  +  wt(xt)  (!) 

y,  =ht(xt)  +  vt(xt) 
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The  weights  are  updated  at  each  time  step  by  multiplying  by  the  conditional  probability  of  the  observation 
y  given  the  true  state  x.  This  probability  is  estimated  usually  by  the  distance  between  the  particles  that 
make  up  the  probability  density  of  x  and  the  observation: 

W®  =  w®,  p(y,  I X®) 

pix.l-Vw'Vsis  X®)  (2) 

i=l 

Also,  the  weights  are  normalized  at  each  time  step: 


w[°  <—  w[ l) /n  n  = 

i=l 


(3) 


Numerical  classification  is  done  via  support  vector  machines  (SVM).  Support  vector  machines  are  not  a 
statistical  technique,  but  future  work  will  involve  including  Bayesian  classifiers  (along  with  other 
classifiers)  in  the  toolbox  of  agent  classifiers.  Solving  a  support  vector  machine  consists  of  solving  an 
optimization  problem.  Given  a  collection  of  labeled  training  data  (Xj,y;),  i=l,2,...,n,  y;  e  {+1,-1},  x;e  Rd: 


minimize  V^Hwlp+CS^i, 
subject  to  yi(wTcj)(Xi)+b)>l-£,i,  t,>0, 


(4) 


where f  represent  slack  variables,  C  is  an  error  penalization  parameter,  and  w  and  b  define  a  linear 
regressor  in  the  feature  space  (the  range  space  of  the  function  (j>).  Support  vector  machine  projects  data 
into  a  higher  dimensional  space  where  linearly  separation  between  the  data  is  possible.  Then  the 
hyperplane  wTi|)(xi)+b=0  is  derived,  maximizing  the  margin  of  separation.  The  hyperplane  can  be 
represented  as: 


XA.iyi<))(xi)-<t)(x)=0, 


(5) 


where  the  A,;  are  Lagrange  multipliers.  t))(Xi)  (j)(Xj)  is  called  the  kernel  function  K(Xj,Xj). 


3.0  Categorical  Methods 

Category  theory,  higher  category  theory,  and  topos  theory  supply  a  suitable  model  for  sensor  networks.2'3 
Category  theory  views  mathematical  structures  at  a  higher  level  of  abstraction.  Category  theory  is  a  way 
of  doing  mathematics  emphasizing  mappings  more  than  objects,  processes  more  than  things.  Its 
advantage  over  other  modeling  formalisms  lies  in  its  generality  and  extensive  theory.  Higher  category 
theory  expands  category  theory  into  higher  dimensions,  allowing  higher  levels  of  mappings.  Topos  theory 
was  developed  both  through  the  application  of  category  theory  to  logic  and  algebraic  topology  and 
algebraic  geometry.  We  rely  on  Goldblatt  for  our  descriptions  of  category  theory  and  categorical  logic.4 
A  category  is  a  collection  of:  1)  arrows;  2)  objects;  3)  assignments  to  each  arrow  a  domain  (object  Oi) 
and  a  codomain  (object  O2)  (Figure  3);  4)  a  composition  operation  for  arrows  that  is  associative  (Figure 
4);  and  5)  an  identity  arrow  (1G)  for  each  object  (O)  (Figure  3). 
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Figure  3.  Diagram  of  domain  and  codomain  (left),  and  identity  arrow  (right). 


Figure  4.  Diagram  of  associativity. 


An  object  O  in  category  C  is  initial  (terminal)  if  there  is  only  one  arrow  from  (to)  O  to  (from)  every  object 
in  C.  A  product  of  objects  X  and  Y  is  an  object  Xx  Y  and  arrows  from  XxY  to  X  and  Y  such  that  for  any 
object  Z  with  arrows  to  X  and  Y  there  is  only  one  arrow  from  Z  to  XxY  such  that  the  following  diagram 
commutes — different  paths  of  arrows  from  one  object  to  another  produce  equivalent  mappings.  (See 
Figure  5). 


Figure  5.  Diagram  of  product. 


Given  arrows  a  and  b  from  object  X  to  Y,  an  equalizer  is  an  arrow  e  to  X  such  that  a°e=b°e  (Figure  6). 


Figure  6.  Diagram  of  equalizer. 
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A  cone  for  a  diagram  is  an  object  O  and  an  arrow  from  O  to  each  object  in  the  diagram.  O  is  a  limit  if  for 
any  other  cone  originating  from  object  O’,  there  is  exactly  one  arrow  from  O’  to  O  such  that  the  following 
diagram  (Figure  7)  commutes. 


Figure  7.  Diagram  of  a  cone. 


Category  theory  can  be  applied  to  logic.  Classical  propositional  logic  consists  of  sentences  a,  b,  that  are 
evaluated  as 

TRUE=1={0}  (0  is  the  null  set;  1  is  the  set  with  the  null  set  as  its  only  element.)  (6) 

or 

FALSE=O=0.  (7) 

These  sentences  are  combined  with  the  following  connectives:  conjunction  aflb,  disjunction  aUb, 
negation  ~a,  and  implication  =>.  The  truth  values  for  new  combined  sentences  are  given  by  the  standard 
truth  tables.  In  an  axiomatic  system,  truth  is  instead  assigned  to  a  collection  of  primary  sentences  called 
axioms.  New  true  sentences  can  then  be  built  up  through  deductive  rules.  In  the  category  of  sets,  the  set 

2={O,1}={0,{0}}.  (8) 

Conjunction  is  the  characteristic  function  of  {<1 , 1>} ,  a  subset  of  2X2.  Negation  is  the  characteristic 
function  of 

i =m,  (9) 

a  subset  of  2.  Implication  is  the  characteristic  function  of 

<={<0,0>,<0,1>,<1,1>}, 

a  subset  of  2X2.  True  is  the  map  from  {0}  to  {0, {0} }  which  takes  1  to  1, 

True(l)=l. 

False  is  the  map  from  {0}  to  {0,  {0} }  which  takes  1  to  0, 

False(l)=0.  (12) 

In  order  to  include  decision  fusion  in  our  system,  we  utilize  topos  theory.  A  topos  is  a  category  that  has: 

1)  all  finite  limits;  2)  exponentiation;  3)  a  subobject  classifier.  A  category  has  all  finite  limits  when  all 
finite  diagrams  have  a  limit.  Exponentiation  and  the  subobject  classifier  can  be  illustrated  through  their 
set  theoretical  parallels.  For  set  S,  2s  is  the  set  of  mappings  from  S  to  {0,1}.  For  any  subset  s  of  S,  the 
standard  characteristic  function  is  equivalent  to  the  mapping  of  2s  which  sends  each  element  of  s  to  1  and 
each  non-element  to  0.  Thus,  2  is  the  subobject  classifier  for  the  topos  of  sets.  As  will  be  shown,  topos 
theory  is  a  methodology  for  representing  indeterminacy. 5  A  functor  is  a  mapping  between  categories  that 
preserves  structure.  Objects  map  to  objects.  Arrows  map  to  arrows.  The  identity  arrows  and  arrow 
composition  are  preserved.  A  contravariant  functor  is  a  functor  that  reverses  the  direction  of  arrows.  It 


(10) 

(11) 
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maps  domains  to  codomains  and  vice  versa.  A  presheaf  is  a  contravariant  functor  from  a  category  to  the 
category  of  sets.  The  category  of  presheaves  over  a  category  is  a  topos. 

A  higher  dimensional  category  is  a  category  whose  objects  are  the  arrows  of  a  category  C,  and  the  arrows 
are  mappings  between  them.6  Figure  8  shows  some  examples  of  arrows  in  a  2-dimensional  category. 
Fligher  categories  can  also  have  a  chain  of  multiple  arrows  as  a  source  (Figure  9).  The  process  can  be 
continued  into  higher  dimensions  (Figure  9). 


3-dimensional  category  (right).  The  black  3-dimensional  arrow  has  as  source  and  target,  the  white 
2-dimensional  arrows. 

Category  theory  has  been  applied  to  modeling  agents,  although  not  sensor  agents.  Di  Marzo  et  al.  have 
described  a  formalization  of  multi-agent  systems  in  category  theory. 7  Although  rudimentary,  their  ideas 
parallel  some  of  our  own,  especially  by  suggesting  topos  theory  as  a  model  of  indeterminacy.  They 
describe  categories  of  computational  units,  agents,  services,  and  multi-agent  systems.  Functors  are  then 
defined  between  these  categories.  The  robustness  of  category  theory  is  that  one  can  diagram  one’s  system 
in  a  multitude  of  ways  and  then  have  access  to  advanced  mathematical  tools  to  manipulate  it. 

4.0  Putting  it  All  Together 

Our  agent  model  is  at  this  point  straightforward.  We  intend  to  later  add  greater  sophistication.  Its  current 
purpose  is  to  provide  a  modeling  framework  within  which  to  develop  our  categorical  system.  Agents 
types  are:  sensor  agents  (basic  fixed  sensors — currently  low  level  image  processing  takes  place  within  the 
sensor  agent,  although  we  intend  to  test  adding  functionality  with  a  general  image  processing  agent  type); 
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classifying  agents  (mobile  agents  which  visit  the  individual  sensors,  apply  their  respective  classifying 
algorithms,  and  report  to  a  higher  level — currently  the  only  differences  between  individual  classifying 
agents  are  the  parameters  of  their  SVMs,  but  later  we  intend  to  include  a  vast  diversity  of  classifiers); 
coordinating  agents  (these  agents  are  assigned  groups  of  sensors  to  monitor — they  receive  input  from  the 
classifying  agents  but  can  also  access  raw  data  from  the  sensor  agents  when  necessary);  and  decision 
agents  (these  agents  receive  information  from  the  coordinating  agents,  assign  tasks  to  the  coordinating 
agents,  and  report  their  decisions  to  the  human  monitor — currently  tracking  is  done  at  this  level,  but  we 
are  experimenting  with  incoiporating  the  tracking  algorithm  into  other  types  of  agents).  The  human 
monitor  is  the  pinnacle  of  the  system.  He/she  has  access  to  all  agent  activity  but  generally  deals  with  the 
decision  agents  while  referring  upon  possible  target  identification  to  the  sensor  agents.  The  human 
monitor  adds  the  vital  element  of  subjectivity,  making  it  a  symbiotic  subjective  system.  As  an  interesting 
aside,  it  could  be  possible  with  the  success  of  the  current  research  to  supplant  the  human  monitor  with  an 
animal  in  some  situations.  Figure  10  depicts  a  diagram  the  system. 


Figure  10.  Diagram  of  the  agent  system,  (figure  at  center  represents  “human  in  the  loop.”) 


Currently,  agents  interact  through  a  simple  blackboard  system.  Each  agent  has  a  multi-sheet  spreadsheet 
file  that  it  writes  to.  Agents  gain  information  through  accessing  other  agents’  files.  So  in  fact,  a  mobile 
agent’s  movement  consists  solely  of  reading  successive  files.  A  sensor-agent’s  sensor  input  is  written  as  a 
multi-dimensional  matrix  on  the  spreadsheet.  Functions  exist  for  manipulating  data,  reading  spreadsheets, 
performing  classifications,  and  fusing  data  and  classifications.  For  the  sensor  agents,  these  functions 
include  feature  extraction  with  the  biologically-based  filters.  Each  agent  is  assigned  a  type,  and  each 
function’s  input  requires  a  specific  type,  but  otherwise  no  protocol  exists  for  restricting  permission  to 
access  agent  files.  Of  course,  permissions  will  be  added  as  the  system  evolves.  Classifiers  are  presently 
strictly  SVMs  with  various  parameter  formulations,  although  this  too  will  be  expanded  to  include 
Bayesian  and  other  numerical  classifiers — but  also  logical  classifiers.  The  value  of  category  theory  is  that 
it  enables  fusion  of  these  disparate  types  of  information. 


Combined  Statistical,  Biological  and  Categorical  Models  for  Sensor  Fusion 


8 


We  currently  have  defined  two  categories  in  our  system.  First,  we  have  a  category  of  agents,  with 
subcategories  for  each  agent  type:  classifying,  sensor,  coordinating,  and  decision.  Each  individual  agent  is 
an  object  of  a  category.  The  arrows  are  the  functions  between  agents.  Second,  we  have  a  process 
category,  whose  objects  are  the  agent  functions  but  also  the  image  processing  operations  and  the 
classifying  and  fusion  operations — which  correspond  to  an  object  with  an  arrow  leading  back  to  itself. 
These  separate  operations  can  be  considered  subcategories.  With  both  agents  and  processes,  it  has  not  yet 
been  fully  determined  to  what  degree  we  will  work  with  the  subcategories  as  separate  categories  or 
subsume  them  under  the  general  categories.  Note  that  the  processes  category  is  in  fact,  a  2-dimensional 
category. 

Graph  theory  adapts  well  to  the  study  of  sensor  networks.  It  also  generalizes  easily  into  category  theory, 
with  nodes  in  a  directed  graph  corresponding  to  objects  in  a  category  and  directed  edges  corresponding  to 
arrows  between  the  objects.  We  use  the  graph  theoretical  formulation  of  sensor  networks.  However,  in 
order  to  apply  topos  theory,  we  work  with  the  topos  of  graphs,  8  which  is  a  presheaf  topos.  Here  a  topos 
consists  of  a  base  space,  objects  projecting  on  to  the  base  space  (individual  directed  graphs),  and  a 
subobject  classifier.  The  base  space  is  the  category  with  objects  (Nodes  and  Arcs)  and  arrows  (Source  and 
Target),  Ns,t— >A,  with  a  Source  arrow  from  N  to  A  if  node  N  is  the  source  of  arc  A  and  a  Target  arrow  if 
node  N  is  the  target  of  arc  A.  Then  any  directed  graph  (object  in  the  topos)  is  a  projection  onto  the  base 
space  (Figure  1 1).  For  any  subgraph  of  that  graph,  the  subobject  classifier  determines  membership  in  the 
subobject,  in  other  words,  the  “truth”  of  membership  in  the  subobject  is  not  either/or.  The  logic  of  a 
presheaf  topos  is  not  necessarily  Boolean.  The  law  of  the  excluded  middle  might  not  apply.  This  is  how 
topos  theory  models  indeterminacy. 


Figure  1 1 .  Sample  graph  over  base  space. 

Note:  Membership  in  the  subgraph  (enclosed  by  dotted  line)  is  determined  by  mappings  into  the  subobject  classifier.  Nodes 
inside  the  subobject  map  to  Node  1  (true) — that  outside  map  to  Node  0  (false).  Arcs  map  to  various  arcs  of  the  subobject 
classifier  depending  on  whether  they  are  inside  or  outside  (or  neither)  of  the  subobject. 

This  methodology  can  be  applied  to  sensor  fusion  in  two  ways.  The  first  begins  with  an  individual  sensor 
agent.  Feature  extractions  are  the  presheaves,  and  the  subobject  classifier  is  the  classifying  agent,  as 
illustrated  in  Figure  12.  Another  method  would  apply  the  theory  to  the  entire  sensor  network,  with 
presheaves  identified  as  individual  sensor  network  configurations  over  the  sensor  network  base  space. 
This  second  method  is,  in  effect,  applying  higher  category  theory  to  the  problem.  Fusion  is  then  modeled 
in  our  system  by  pullbacks  (a  type  of  product).9 
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Figure  12.  Simple  Sensor  Fusion  Topos  Model. 


5.0  Conclusion 

In  this  paper,  we  have  devised  a  system,  in  fact  a  test  bed,  for  representing  sensor  fusion  and  conducting 
sensor  fusion  experiments.  We  have  adapted  state  of  the  art  lower  level  fusion  techniques  and  combined 
them  with  agent  models  and  category  theory.  We  are  confident  that  agency  within  the  category  theory 
framework  prove  fruitful  in  amalgamating  diverse  sensors  and  sensor  modalities — and  both  various 
numerical  and  logical  classifiers.  In  addition,  with  topos  theory  we  have  shown  a  path  toward  effectively 
formalizing  the  decision  process,  and  optimizing  the  placement  of  humans  in  the  system — all  toward 
creating  a  hybrid,  subjective,  sensor  network. 
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